Navigation apparatus and surgical operation image acquisition/display apparatus using the same

ABSTRACT

A navigation apparatus comprises a navigation-related information generating section and a display section. The navigation-related information generating section measures the position and orientation of an object and a target in a three-dimensional space and generate navigation-related information to be used for navigating the object toward the target. The display section displays the navigation-related information generated by the navigation-related information generating section in any of different modes depending on the relationship of the position and orientation of the object and that of the target. A surgical operation image acquisition/display apparatus comprises an observation section, an image display section and a specifying section. The observation section includes a plurality of observation sections whose position and orientation is modifiable. The image display section is adapted to alternatively display any of the images obtained by the observation sections or synthetically combine and display the combined images. The specifying section specifies the image to be displayed to the image display section according to the position and orientation of the observation section.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based upon and claims the benefit of priorityfrom the prior Japanese Patent Applications No. 11-089405, filed Mar.30, 1999; and No. 11-163964, filed Jun. 10, 1999, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND OF THE INVENTION

[0002] This invention relates to a navigation apparatus and, moreparticularly, to a navigation apparatus adapted to modifynavigation-related information according to the relative position andorientation of the object of navigation and the target within athree-dimensional space.

[0003] This invention also relates to a surgical operation imageacquisition/display apparatus and, more particularly, to an operationimage acquisition/display apparatus adapted to acquisition and displayimages of a plurality of observation systems used in surgicaloperations.

[0004] Various navigation apparatus have been proposed for applicationsin the field of surgical operations, including those disclosed in Jpn.Pat. Appln. KOKAI Publication Nos. 9-173352 and 10-5245.

[0005] The medical navigation system disclosed in Jpn. Pat. Appln. KOKAIPublication No. 9-173352 is adapted to display information (profileinformation, medical image information) on the desired part of theobject of examination specified by a specifying section for specifying adesired part of the object of examination.

[0006] It is also adapted to display video information obtained by anappearance imaging section, profile information on the profile measuredby a profile measuring section and medical image information obtained bya medical image acquisition section on an image display section in anoverlaid way.

[0007] The surgical operation assisting apparatus disclosed in Jpn. Pat.Appln. KOKAI Publication No. 10-5245 is adapted to display the currentposition of the surgical instrument being used in a surgical operationand the blood vessel located closest to the instrument on a tomographicimage of the area of surgical operation in an overlaid manner by usingthe image data on the tomographic image, the surgical instrument beingused, blood vessel detection section for detecting the blood vessellocated closest to the surgical instrument, a position detection sectionfor detecting the current position of the surgical instrument, anarithmetic computing section for computationally determines the positionof the front end of the surgical instrument and the direction in whichthe surgical instrument is inserted, an image selection section forselecting the image data on the image being acquired for the area wherethe front end of the surgical instrument is located and an imagesynthesizing section for synthetically combining the image selected bythe image selection section and a predetermined pattern indicating thefront end of the surgical instrument in an overlaid manner.

[0008] The above described arrangement is intended to allow the operatorto visually confirm the position of the front end of the surgicalinstrument inserted into the body of the patient on the tomographicimage being displayed.

[0009] However, the medical navigation system and the surgical operationassisting apparatus as disclosed in the above patent documents areaccompanied by the following problems.

[0010] As for the medical navigation system disclosed in Jpn. Pat.Appln. KOKAI Publication No. 9-173352, it simply displays information ona desired part of the object of examination specified by the section forspecifying a desired part of the object of examination and it isdifficult to navigate the section to the part desired by the user.

[0011] Additionally, this known medical navigation system provides adifficulty with which the surgeon realizes distances in the perspectiveof displayed information in the direction connecting the eyes of thesurgeon and the display screen that is perpendicular to the latter.

[0012] Furthermore, this known medical navigation system provides anadditional difficulty with which the surgeon determines the route ofnavigation on the basis of the displayed information when both theobject of examination and the section for specifying the desired part ofthe object of examination are located at respective positions that arefound within the measurable area but outside the displayable area of thesystem.

[0013] The surgical operation assisting system disclosed in Jpn. Pat.Appln. KOKAI Publication No. 10-5245 is accompanied by a problem ofcumbersomeness that the user has to be constantly aware of the distancebetween the position of the front end of the surgical instrument on thedisplayed tomographic image and the position of the detected bloodvessel in order to know the distance between the blood vessel and thesurgical instrument.

[0014] In recent years, micro-surgery has become popular as a result ofthe development of both surgical techniques and surgical instruments.

[0015] In micro-surgery, generally a surgical microscope is used toobserve an enlarged view of the area of surgical operation.

[0016] Particularly, in the field of cranial nerve surgery andotorhinolarygology, there arise occasions frequently where the area ofoperation can hardly be observed because it is at the so-called deadangle if the surgical microscope is handled elaborately when the area islocated deep in the body.

[0017] For observing an area at such a dead angle, normally a mirror oran endoscope is used.

[0018] When using an endoscope for micro-surgery, it has to manipulatedand placed accurately at the right position that is located deep in thebody having an exquiarealy complicated three-dimensional structurebecause the area of operation is always at the dead angle of thesurgical microscope.

[0019] The manipulation has to be conducted carefully by the operator,while observing it through the surgical microscope so that any normaltissues of the patient would not be inadvertently damaged by theendoscope and, at the same time, the area of operation has to bevisually confirmed by means of the endoscope.

[0020] While manipulating the endoscope, the operator has to selectinstantaneously either the image taken by the surgical microscope or theimage acquired by way of the endoscope as object of observation and theselection has to be correct.

[0021] As an attempt for aiding a surgeon manipulating the endoscope,Jpn. Pat. Appln. KOKAI Publication No. 5-203881 proposes an integratedimage system comprising a plurality of CCD cameras connected torespective observation systems, each including a surgical microscope, anendoscope and other instruments, a CCD camera controller for controllingthe operation of selectively using any of the observation systems and aview finder controller so that the user may select any of theobservation systems by means of the CCD camera controller in the courseof the ongoing surgical operation.

[0022] Jpn. Pat. Appln. KOKAI Publication No. 7-261094 discloses asurgical microscope with which the user can switch from the image of thesurgical microscope to that of the endoscope or vice versa or overlayone on the other whenever necessary.

[0023] However, with the known technique disclosed in the abovedescribed Jpn. Pat. Appln. KOKAI Publication No. 5-203881, the operatorhas to carry out the switching or overlaid operation at the cost of asmooth progress of the ongoing surgical operation.

[0024] Additionally, while the above patent document describes that theimage may be switched from one to the other, it does not describespecifically how the switching operation proceeds.

[0025] On the other hand, the known technique disclosed in Jpn. Pat.Appln. KOKAI Publication No. 7-261094 involves the use of a mode switchwith which the surgical operator can switch the display mode whenevernecessary.

[0026] However, it is highly cumbersome for the operator to switch fromthe image of the surgical microscope to that of the endoscope or viceversa when he or she has to place the endoscope in a position deep inthe body of the patient having an exquiarealy complicatedthree-dimensional structure. Additionally, such a switching operationcan obstruct the smooth progress of the surgical operation.

BRIEF SUMMARY OF THE INVENTION

[0027] In view of the above identified problems of the prior art, it istherefore the object of the present invention to provide a navigationapparatus with which the user can easily and visually realize thedistance between a target and an object of navigation by modifying theobtained navigation-related information according to the relativeposition and orientation of the object of navigation and the targetwithin a three-dimensional space and the user can easily obtainnavigation-related information of the type necessary for the user.

[0028] Another object of the invention is to provide an operation imageacquisition/display apparatus adapted to acquisition and display imagesof a plurality of observation systems used in surgical operationswithout requiring the operator to manually switch from one observationsystem to another so that the ongoing surgical operation may proceedssmoothly.

[0029] In the first aspect of the invention, the above first object isachieved by providing a navigation apparatus comprising:

[0030] a navigation-related information generating section forgenerating navigation-related information by measuring the relativeposition and orientation of an object and a target in athree-dimensional space in order to navigate the object to the target;and

[0031] a display section for displaying the navigation-relatedinformation generated by the navigation-related information generatingsection in different modes according to the relative position andorientation of the object and the target.

[0032] Thus, with a navigation apparatus according to the inventionadapted to display navigation-related information in different modesaccording to the relative position and orientation of the object and thetarget within a three-dimensional spatial.

[0033] A navigation apparatus according to the invention will bedescribed hereinafter in terms of the firs and second embodiments. Whilethe above target may normally be a patient, a tumor to be surgicallytreated of a patient or an area of the body of a patient requiringspecial attention during a surgical operation, it is by no means limitedto an existing object of examination and may alternatively be a virtualtarget displayed as a two-dimensional or three-dimensional image of amodel synthesized by using the video information of an existing targetthat is obtained in advance.

[0034] While the above object may normally refer to an endoscope 3, itmay alternatively refer to some other surgical instrument such as asuction pipe or a pair of forceps.

[0035] While the above display section may normally refer to a liquidcrystal monitor, it may alternatively refer to some other videoinformation display such as a CRT display or a head mount display.

[0036] For the purpose of the invention, the above expression “indifferent modes” refers to differences in color, in the thickness ofline, in dimensions and in the density of drawing.

[0037] In the second aspect of the invention, the above second object isachieved by providing a surgical operation image acquisition/displayapparatus comprising:

[0038] an observation section having a plurality of observation sectionand adapted to modify its position and orientation;

[0039] an image display section adapted to alternatively orsynthetically display the images obtained by the plurality ofobservation section of the observation section; and

[0040] an indication section for indicating the images to be displayed.

[0041] Additional objects and advantages of the invention will be setforth in the description which follows, and in part will be obvious fromthe description, or may be learned by practice of the invention. Theobjects and advantages of the invention may be realized and obtained bymeans of the instrumentalities and combinations particularly pointed outhereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

[0042] The accompanying drawings, which are incorporated in andconstitute a part of the specification, illustrate presently preferredembodiments of the invention, and together with the general descriptiongiven above and the detailed description of the preferred embodimentsgiven below, serve to explain the principles of the invention.

[0043]FIG. 1 is a schematic illustration of the first embodiment of theinvention which is a navigation apparatus, showing its configuration;

[0044]FIG. 2 is a schematic illustration of a distance map that can beused for a navigation apparatus according to the invention;

[0045]FIG. 3 is a schematic illustration of a distance map that can beused for a navigation apparatus according to the invention;

[0046]FIG. 4 is a schematic illustration of the relationship betweendata on an object of examination and the object of examination itself;

[0047]FIG. 5 is a schematic illustration of a coordinate transformationmatrix for correlating data on an object of examination and the objectof examination itself;

[0048]FIG. 6 is a schematic illustration of a coordinate transformationmatrix for transforming a coordinate system defined by a sensing platefitted to an endoscope into a coordinate system to be used by a cameramodel expressing the optical system of the endoscope and a coordinatetransformation matrix for transforming the coordinate system of a cameramodel into the coordinate system of a liquid crystal monitor;

[0049]FIG. 7 is a schematic illustration of a coordinate transformationmatrix for transforming a coordinate system defined by a sensing platefitted to the head of an object of examination into the coordinatesystem defined by a sensing plate fitted to an endoscope.

[0050]FIG. 8 is a schematic illustration of a transformation using aplurality of coordinate transformation matrices for transforming data ona target area into positional data on a liquid crystal monitor;

[0051]FIGS. 9A through 9D schematically illustrate examples of imagesthat may be displayed on a liquid crystal monitor, of which FIG. 9A isan image obtained by overlaid a wireframe image as navigationinformation on an image obtained by means of the optical system of anendoscope, FIG. 9B is an image obtained by overlaid an internaltomographic image of three-dimensional volume data as navigationinformation on an image obtained by means of the optical system of anendoscope, FIG. 9C is an image obtained when no target area is foundwithin the effective area of measurement of an endoscope and FIG. 9D isan image obtained when the apparatus is inoperative for measurement;

[0052]FIG. 10 is a flow chart of a display operation for displaying animage as shown in FIG. 9C;

[0053]FIG. 11 is a schematic illustration of the second embodiment ofthe invention which is also a navigation apparatus, showing itsconfiguration;

[0054]FIG. 12 is a schematic illustration of an example of an imagedisplayed by the second embodiment and a coordinate transformationmatrix that can be used for the display;

[0055]FIG. 13 is a schematic illustration of an operation of modifyingthe thickness of lines of an orthogonally projected image of anendoscope as a function of the relative distance between the target areaand the front end of an endoscope;

[0056]FIG. 14 is a schematic block diagram of the third embodiment ofthe invention which is a surgical operation image acquisition/displayapparatus, showing its configuration;

[0057]FIGS. 15A through 15F are schematic illustrations of a pluralityof display modes that can be realized by the video mixer 143 of FIG. 14;

[0058]FIG. 16 is a schematic illustration of an operation of correlatingdata on the operation area of a patient 146 and data on thecharacteristic points of a model data coordinate system m;

[0059]FIG. 17 is a schematic illustration of a mode of computationallyobtaining a coordinate transformation matrix pHe for transforming thepatient coordinate system p defined by the sensing plate 145 b fitted tothe head of a patient 146 to the endoscope coordinate system e definedby the sensing plate 145 c fitted to an endoscope 142;

[0060]FIG. 18 is a flow chart of the operation of the image controller147 of FIG. 14; and

[0061]FIG. 19 is a flow chart of the operation of the image controller147 of the fourth embodiment of the invention which is also a surgicaloperation image acquisition/display apparatus.

DETAILED DESCRIPTION OF THE INVENTION

[0062] Reference will now be made in detail to the presently preferredembodiments of the invention as illustrated in the several views of theaccompanying drawing, in which like reference numerals designates likeor corresponding parts.

[0063] (Embodiment 1)

[0064]FIG. 1 is a schematic illustration of the first embodiment of theinvention which is a navigation apparatus, showing its configuration.

[0065] Referring to FIG. 1, object of examination 1, or patient, islying flat on an operating table, facing upward.

[0066] A hard sensing plate 2 carrying three LEDs for emitting infraredrays that are arranged at the respective corners of a triangle issecurely fitted to the head of the object of examination 1 in such a waythat its position and orientation relative to the head would not changeeasily.

[0067] Another hard sensing plate 4 carrying three LEDs for emittinginfrared rays arranged at the respective corners of a triangle issecurely fitted to an endoscope 3.

[0068] The LEDs arranged on the sensing plate 2 and those arranged onthe sensing plate 4 do not change their positional relationships.

[0069] The positions of the LEDS of each of the sensing plates 2 and 4are observed and determined in advance in terms of the coordinate systemdefined on the sensing plate and stored in sensor information storagesection 5 as LED definition data.

[0070] The sensor information storage section 5 is connected to sensorcontrol section 6.

[0071] Then, image acquisition type sensor assembly 7 is arranged at aposition where the sensing plates 2 and 4 are found within its effectivearea of measurement.

[0072] Then, a three-dimensional position and orientation measuringsection is established as the sensing plates 2 and 4 and the sensorassembly 7 are connected to sensor control section 6.

[0073] The three-dimensional position and orientation informationobtained by the three-dimensional position and orientation measuringsection is sent to navigation-related information control section 8.

[0074] The information including profile information and internaltomographic image information on the object of examination, the tumorthereof to be surgically treated and the parts thereof requiring specialattention during a surgical operation and obtained in advance bymeasurement using CT and/or MRI is divided into low resolutioninformation (e.g., for a resolution level of 32×32×32 voxels), mediumresolution information (e.g., for a resolution level of 128×128×128voxels) and high resolution information (e.g., for a resolution level of512×512×512) and then transformed into wireframe three-dimensional modeldata 10 (high resolution wireframe three-dimensional model data 10 a,medium resolution wireframe three-dimensional model data 10 b, lowresolution wireframe three-dimensional model data 10 c) andthree-dimensional volume data 11 (high resolution three-dimensionalvolume data 11 a, medium resolution three-dimensional volume data 11 b,low resolution three-dimensional volume data 11 c) and stored in thenavigation-related information storage section 9 as data.

[0075] The navigation-related information storage section 9 additionallystores in advance as distance map.

[0076] As shown in FIGS. 2 and 3, a distance map 12 contains athree-dimensional array having values representing the shortestdistances from the surface of the target area (the object ofexamination, the tumor to be surgically treated or the parts of the bodyrequiring special attention during a surgical operation), the affixednumbers of the array being variables corresponding to thethree-dimensional positional coordinate system of the space where thetarget area is located.

[0077] For example, when the smallest unit of division is 0.1 mm, a{fraction (1/10)} of an index number represents a correspondingcoordinate value as expressed in terms of millimeter.

[0078] Assume that such a distance map is prepared for each target areain advance by means of a distance map preparing computer and stored inthe navigation-related information storage section 9 as data.

[0079] Note that all the wireframe three-dimensional model data 10, thethree-dimensional volume data 11 and the distance map 12 are subjectedto a coordinate transforming operation so that they are be expressed interms of a same coordinate system.

[0080] Then, the image obtained by way of the optical system of theendoscope 3 is taken into the navigation-related information controlsection 8 by way of a camera control unit and an image input board (notshown).

[0081] The navigation-related information generated by thenavigation-related information control section 8 is display to the useron information display section, which is a liquid crystal monitor 13.

[0082] As shown in FIG. 4, data on the object of examination 1 and theobject of examination 1 itself are correlated by measuring thecoordinate value m of each characteristic point on the data and thecoordinate values p of the corresponding characteristic point as definedby the sensing plate 2 and computing a coordinate transformation matrix(pHm) 14.

[0083] The coordinate transformation matrix (pHm) 14 is stored in theabove navigation-related information storage section 9.

[0084] As shown in FIG. 5, a coordinate transformation matrix is a 4-rowand 4-column matrix comprising a rotational component R representing arotary motion in a three-dimensional space, a translational component Trepresenting a translation in the three-dimensional space and a constantcomponent.

[0085] Additionally, as shown in FIG. 6 a coordinate transformationmatrix (cHe) 15 for transforming the coordinate system defined by thesensing plate 4 into the coordinate system to be used by a camera modelexpressing the optical system of the endoscope 3 and a coordinatetransformation matrix (f_ctos) 16 for transforming the camera modelcoordinate system into the coordinate system on the actual liquidcrystal monitor 13 are also determined and stored in thenavigation-related information storage section 9.

[0086] Now, the operation of the first embodiment of navigationapparatus according to the invention and having the above describedconfiguration will be discussed below.

[0087] During the operation of the navigation apparatus, the sensorcontrol section 6 that is a component of the three-dimensional positionand orientation measuring section measures the three-dimensionalposition of each of the LEDs that are emitting infrared rays of thesensing plates 2 and 4 and then computationally determines thethree-dimensional position and orientation information of each of thesensing plates 2 and 4 in terms of the coordinate values of the originalpoint of the space defined by the sensing plate 4 on thethree-dimensional space defined by turn by the sensing plate 2 and thevalues of the unit vectors along the X, Y and Z axis of the spacedefined by the sensing plate 4 by using the LED definition data storedin the sensor information storage section 5.

[0088] Then, as shown in FIG. 7, the coordinate transformation matrix(pHe) 17 from the sensing plate 2 attached to the head of the object ofexamination 1 to the sensing plate 4 attached to the endoscope 3 iscomputationally determined on the basis of the obtainedthree-dimensional position and orientation information.

[0089] Then, as shown in FIG. 8, the data of target area is converted tothe positional data on the liquid crystal monitor 13, thenavigation-related information control sections 8 generatesnavigation-related information by using the obtained positional databased on the coordinate transformation matrix 17 and the coordinatetransformation matrixes 14, 15 and 16.

[0090] As the image formed by the optical system of the endoscope 3 isinput to the navigation-related information control section 8, thenavigation-related information and the image are displayed on the liquidcrystal monitor 13 in an overlaid manner as shown in FIG. 9A.

[0091] Then, as shown in FIG. 7, the position of the front end of theendoscope 3 is subjected to an operation of coordinate transformation byusing the above described coordinate transformation matrices 14, 15 and17 and the relative distance between the target area and the front endof the endoscope 3 is determined by referring to the distance map 12.

[0092] Then, as shown in FIG. 9A, when the object of examination 1 andthe endoscope 3 are in a measurable state, the relative distance betweenthe target area and the front end of the endoscope 3 is displayed on theliquid crystal monitor 13 as the distance to the tumor in terms of thelength of a bar 30 and a numerical value 31.

[0093] In a surgical operation using an endoscope 3, the endoscope 3 hasto be inserted toward the tumor from the outside of the object ofexamination 1, paying attention to the parts that should not be damaged,and then the tumor has to be surgically treated.

[0094] When the endoscope 3 is located outside the object of examination1, the model image of the target area is generated as a profiled,wireframe image 18 as shown in FIG. 9A.

[0095] In this embodiment, the color and the thickness of the lines ofthe wireframe image 18 are made to vary as a function of the relativedistance between the front end of the endoscope 3 and the surface of thetarget area as determined in a manner as described above.

[0096] The color and the width of the bar 30 and those of the numericalvalue 31 showing the distance to the tumor are also made to vary alongwith those of the background.

[0097] For instance, when the relative distance is equal to or greaterthan 10 mm, the color of the lines of the wireframe image 18 may be blueand the thickness of the lines may be equal to 1 pixel while both thecolor of the bar 30 representing the distance and that of the backgroundof the numerical value 31 may be equally blue and the width of the bar30 may equal to 20 pixels.

[0098] When, on the other hand, the relative distance is equal to orgreater than 0 mm and smaller than 10 mm, the color of the lines of thewireframe image 18 may be yellow and the thickness of the lines may beequal to 2 pixels while both the color of the bar 30 representing thedistance and that of the background of the numerical value 31 may beequally yellow and the width of the bar 30 may equal to 30 pixels.

[0099] If the front end of the endoscope 3 is inserted by a distanceequal to or greater than 0 mm and smaller than 10 mm, the color of thelines of the wireframe image 18 may be purple and the thickness of thelines may be equal to 2 pixel while both the color of the bar 30representing the distance and that of the background of the numericalvalue 31 may be equally purple and the width of the bar 30 may equal to30 pixels.

[0100] In this way, when the front end of the endoscope 3 traveled by apredetermined distance, both the color of the wireframe image 18 and thethickness of the lines of the wireframe image 18 may be made to changeso that the user can visually recognize the distance between the surfaceof the target area and the front end of the endoscope 3.

[0101] Additionally, the wireframe image 18 of an area requiring specialattention may be drawn with thick lines when the endoscope 3 is locatedclose to the area and separated therefrom by a distance smaller than apredetermined value so that the user may visually recognize that theendoscope 3 is too close to the area.

[0102] For instance, when the reference value of the distance map 12 forthe area requiring special attention is less than 10 mm, the lines ofthe area requiring special attention of the wireframe image 18 may bemade five times thicker than before.

[0103] The denseness or coarseness of the wireframe image 18 that isdrawn in correspondence to the relative distance between the endoscope 3and the surface of the target area is also made to vary.

[0104] More specifically, a set of more detailed wireframethree-dimensional model data 10 a will be selected as the relativedistance is reduced, whereas a set of more scarce wireframethree-dimensional model data 10 c will be selected as the relativedistance is increased.

[0105] For instance, high resolution wireframe three-dimensional modeldata 10 a will be used when the distance to the target area is less than30 mm and medium resolution wireframe three-dimensional model data 10 bwill be used when the distance to the target area is between 30 mm and100 mm, whereas low resolution wireframe three-dimensional model data 10c will be used when the distance to the target area is greater than 100mm.

[0106] With this arrangement, the problem of the prior art that coarsewireframe three-dimensional model data have to be used to reduce thetime until the completion of drawing a wireframe image in order to savetime when the endoscope is approaching the target area whereas densewireframe three-dimensional model data are used to unnecessarily consumetime before the completion of drawing a wireframe image when theendoscope is remote from the target area is successfully eliminated anda required level of detail and drawing rate can be realized depending onthe distance between the endoscope and the target area.

[0107] Additionally, when the endoscope 3 is inserted into the object ofexamination 1 by using the above described embodiment, the model imageof the object of examination 1 drawn by the embodiment is switched fromthe wireframe image 18 to the internal tomographic image 19 obtained byusing the three-dimensional volume data of the object of examination 1as shown in FIG. 9B depending on the relative distance between theendoscope 3 and the outermost zone of the target area.

[0108] The relative distance between the front end of the endoscope 3and the surface of the target is determined in a manner as describedabove.

[0109] For instance, the model image of the object of examination may beswitched from the wireframe image 18 to an internal tomographic image 19obtained by using the three-dimensional volume data of the object ofexamination that reflect the viewing direction of the endoscope.

[0110] As a result of this switching operation, the user can easilyacquire the internal tomographic image 19 that is very important afterthe insertion of the endoscope 3 in stead of the wireframe image 18 ofthe object of examination that becomes unnecessary after the insertionof the endoscope 3 without being required to carrying out the switchingoperation by him- or herself.

[0111] Thus, the navigation-related information displayed to the userwhen the endoscope 3 is inserted into the object of examination includesthe internal tomgraphic image obtained by using the three-dimensionalvolume data of the object of examination 1 and the wireframe image 20 ofthe area requiring special attention.

[0112] As the endoscope 3 is brought close to the target and separatedfrom the latter by a distance smaller than a predetermined value underthis condition, not only the drawing attributes of the wireframe image20 but also the color of the internal tomographic image 19 drawn byusing the three-dimensional volume data are made to change.

[0113] If, on the other hand, the target is not found within theeffective area of measurement of the endoscope 3, arrow 21 indicates thedirection in which the target area will be found as shown in FIG. 9C.

[0114] If the model image is found within the drawable (displayable)range or not can be determined by checking if the coordinate of each andevery point of the model image as computed when drawing the model imageis found as a point on the monitor to be used for displaying the image.

[0115] Referring to FIG. 10, the coordinate of the model image istransformed into the coordinate of the display screen (Step S1) and itis determined if the coordinate of a transformed point is found withinthe displayable range of the display screen (Step S2).

[0116] If the coordinate is found within the displayable range, themodel image is displayed on the screen (Step S3).

[0117] If, on the other hand, the coordinate is not found within thedisplay range, the coordinate of a representative point of the modelimage is transformed into the corresponding coordinate of the displayscreen (Step S4) and, at the same time, the coordinate of the front endof the endoscope 3 is transformed into the corresponding coordinate ofthe display screen (Step S5). Then, the distance and the direction ofthe line connecting the representative point of the model image and thefront end of the endoscope 3 are computationally determined (Step S6).

[0118] Thus, the relative distance and the relative direction of theline connecting the center 22 of the endoscope image, or the front endof the endoscope 3, and the target can be determined by transforming thecoordinate values 23 on the model data coordinate system of therepresentative point of the target into the coordinate values on theliquid crystal monitor 13 by means of the above described coordinatetransformation matrices 14, 15, 16 and 17.

[0119] Then, the user can visually comprehend the extent to which theendoscope 3 should be moved to bring the target into the effective areaof measurement of the endoscope 3 by modifying the size of the arrow 21indicating the target area in proportion to the obtained distance.

[0120] When the apparatus is incapable of measuring the distance and thedirection, the sensor control section 6 outputs a message telling theuser that the apparatus is incapable of measuring the distance and thedirection in place of three-dimensional position and orientationinformation.

[0121] Upon receiving this message, the navigation-related informationcontrol section 8 erases the model image being displayed asnavigation-related information and generates character information 28 of“unmeasurable condition” and a yellow pixel frame 29 having a widthequal to 60 pixels, which are then displayed to the user on the liquidcrystal monitor 13 as shown in FIG. 9D.

[0122] Then, the user can easily comprehend that the endoscope 3 or theobject of examination 1 located at a position that makes the intendedmeasurement impossible so that the user can be effectively protectedagainst he risk of operating the endoscope 3 in a wrong way according tonavigation-related information that does not reflect the reality.

[0123] It may be needless to say that the configuration of thisembodiment can be modified and/or altered in various different ways.

[0124] For instance, the area that provides the object of navigation isnot limited to the target area and may alternatively be a plurality ofany areas which are defined by information on the profile of the objectof examination 1 or information on an internal tomographic image.

[0125] It is also possible to carry out a simulation by fitting asensing plate to the head of a virtual object of examination withoutusing an actual object of examination 1.

[0126] The three-dimensional position and orientation measuring sectionmay be made to alternatively comprises a magnetic sensor or a set ofmechanical links and joints, encoders and potentiometers popularly usedin ordinary three-dimensional position and orientation measuringsystems.

[0127] When the object of examination is immobile, it is sufficient tomeasure the three-dimensional position and orientation of the object ofexamination 1, store the information in the sensor information storagesection and utilize it in the computational operation for determiningthe relative three-dimensional position and orientation of the object ofexamination and the object of navigation in advance so that it is onlynecessary to measure the three-dimensional position and orientation ofthe object of navigation when the system is in operation.

[0128] The wireframe for expressing the profile information of thetarget area may be replaced by any known technique for graphicexpression that is popularly used for three-dimensional computergraphics including polygons.

[0129] Alternatively, contour lines and equidistant curves relative tothe viewing direction may be used.

[0130] The endoscope 3 that is the object of navigation may be replacedby a plurality of endoscopes.

[0131] The object of navigation may be a surgical instrument that is notprovided with a section of observation.

[0132] The technique used for determining if the target area is locatedwithin the effective area of measurement or not is limited to the abovedescribed one.

[0133] The technique for determining the relative distance between theobject of navigation and the target area is not limited to the abovedescribed one that uses a distance map and any known technique forcomputationally determining the distance between two points in athree-dimensional space may alternatively be used for determining thedistance between a representative point of the object of navigation anda representative point of the target area for the purpose of theinvention.

[0134] Additionally, the color may be made to change continuously as afunction of the distance in stead of the above described use of a singleboundary value. Alternatively, the color may be made to change stepwiseby providing a plurality of boundary values.

[0135] Similarly, the line thickness may be made to change continuouslyin stead of the above described use of a single boundary value.Alternatively, the line thickness may be made to change stepwise byproviding a plurality of boundary values.

[0136] A situation where there is no navigation-related information tobe displayed may be indicated by making the color to be transparent andthe lines to be practically invisible.

[0137] The density of lines for drawing the model image that varies as afunction of the distance may be made to change continuously on the basisof a single set of data in stead of selectively using a plurality ofsets of data with different levels of density that are provided inadvance as described above.

[0138] The pattern that is displayed when the target area is out of theeffective area of measurement is not limited to the arrow 21.Alternatively, a triangle, a circle, a bar or some other figure may beused. The distance may be expressed by the size of the figure.

[0139] Furthermore, the size of the arrow 21 may be made to varystepwise by using a plurality of preselected values in stead of makingit vary continuously in a manner as described above.

[0140] When a section for determining the density of lines for drawingthe model image of the target area on the basis of a single set of datais provided, it is no longer necessary to store in advance a pluralityof sets of data with different levels of density.

[0141] The navigation-related information indicating a situation wherethe apparatus is incapable of measuring the distance and the directionmay not require both character information 28 and a frame 29. It may besufficient to use only either character information 28 or a frame 29 toconvey the information.

[0142] It may be so arranged that the user can define the color and theline thickness that are used as attributes of the navigation-relatedinformation, the density of lines for drawing the model image, the sizeof the displayed pattern, the boundary values for changing the color andthe line thickness as a function of the distance and the characterstring of the characteristic information 28 indicating that theincapability of measurement of the apparatus.

[0143] [Embodiment 2]

[0144] Now, the second embodiment of the invention, which is anavigation apparatus, will be discussed below.

[0145]FIG. 11 is a schematic illustration of the second embodiment ofthe invention which is also a navigation apparatus, showing itsconfiguration.

[0146] This second embodiment has a configuration same as the abovedescribed first embodiment except the following.

[0147] In this embodiment, the endoscope 3 is not required to pass theimaging information obtained by the optical system to thenavigation-related information control section 8.

[0148] In this embodiment, the navigation-related information storagesection 9 stores in advance vectors 24 for expressing the route alongwhich the endoscope 3 is inserted (minimal invasive route) as data.

[0149] Then, the coordinate values 25 for the front end and the rear endof the endoscope 3 are determined in terms of the coordinate systemdefined by the sensing plate 4 rigidly fitted to the endoscope 3 andstored in the navigation-related information storage section 9.

[0150] Now, the operation of the embodiment having the above describedconfiguration will be described below.

[0151] When the embodiment of navigation apparatus is in operation, thesensor control section 6 measures the three-dimensional position of eachof the LEDs that are emitting infrared rays of the sensing plates 2 and4 and then computationally determines the three-dimensional position andorientation of each of the sensing plates 2 and 4, by using the LEDdefinition data stored in the sensor information storage section 5.

[0152] Then, the coordinate transformation matrix 17 from the sensingplate 2 fitted to the head of the object of examination 1 to the sensingplate 4 fitted to the endoscope 3 is computationally determined on thebasis of the obtained three-dimensional position and orientationinformation.

[0153] Then, the position and the orientation of the endoscope 3 isdetermined in terms of the data on the target area by using thecoordinate transformation matrix 17 and the above described coordinatetransformation matrices 14, 15.

[0154] Then, the navigation-related information control section 8generates a tri-sectional image 26 of the three-dimensional volume data11 including those of the tumor and an orthogonal projection image 27 ofthe endoscope 3 projected on the cross section as navigation-relatedinformation and displays it on the liquid crystal monitor 13.

[0155] If, for instance, the coordinate of a representative point of thetumor is expressed by (260, 180, 280), the tri-sectional image 26 of thethree-dimensional volume data 11 will have a YZ plane with x=260, a ZXplane with y=180 and an XY plane with z=280.

[0156] Then, the relative distance between the surface of the targetarea and the front end of the endoscope 3 is determined by referring tothe above described distance map 12, using the position of the front endof the endoscope 3.

[0157] The thickness of the lines of the orthogonal projection image 27of the endoscope 3 is continuously changed as a function of the distancebetween the surface of the target area and the front end of theendoscope 3.

[0158] Then, the user can easily comprehend a situation where theendoscope 3 is approaching the target area.

[0159] The orientation of the endoscope 3 determined by the coordinatevalues 25 of the position of the front end and that of the rear end ofthe endoscope 3 is compared with the data of the vector indicating thedirection in which the endoscope 3 is to be inserted and, if theorientation is inclined relative to the vector by a predetermined value(e.g., 10 degrees), the color and the line thickness of the orthogonalprojection image 27 of the endoscope 3 will be changed.

[0160] Thus, the user can easily comprehend that the direction in whichthe endoscope 3 is currently inserted is deviating from the direction inwhich it is to be inserted.

[0161] It may be needless to say that the configuration of thisembodiment can be modified and/or altered in various different ways.

[0162] For instance, the area that provides the object of navigation isnot limited to the target area and may alternatively be a plurality ofany areas which are defined by information on the profile of the objectof examination 1 or information on an internal tomographic image.

[0163] It is also possible to carry out a simulation by fitting asensing plate to the head of a virtual object of examination withoutusing an actual object of examination 1.

[0164] The three-dimensional position and orientation measuring sectionmay be made to alternatively comprises a magnetic sensor or a set ofmechanical links and joints, an encoder and a potentiometer popularlyused in ordinary three-dimensional position and orientation measuringsystems.

[0165] When the object of examination is immovable, it is sufficient tomeasure the three-dimensional position and orientation of the object ofexamination 3, store the information in the sensor information storagesection and utilize it in the computational operation for determiningthe relative three-dimensional position and orientation of the object ofexamination and the object of navigation in advance so that it is onlynecessary to measure the three-dimensional position and orientation ofthe object of navigation when the system is in operation.

[0166] The endoscope 3 that is the object of navigation may be replacedby a plurality of endoscopes.

[0167] While the above object of navigation may normally refer to anendoscope 3, it may alternatively refer to some other surgicalinstrument such as a suction pipe or a pair of forceps so long as themechanical profile thereof can be determined by measurement.

[0168] The coordinate of the front end of the endoscope 3 does not needto agree with the actual front end and data may be manipulated to makethe endoscope 3 virtually have an extended front end.

[0169] Alternatively, the coordinate of the front end of the endoscope 3may be defined by means of an operational formula using the extension ofthe front end from the actual front end as parameter so that thecoordinate of the front end may be determined successively on the basisof the extension specified by the user.

[0170] Additionally, the color may be made to change on the basis of asingle boundary value instead of making it change continuously as afunction of the distance in a manner as described above. Alternatively,the color may be made to change stepwise by providing a plurality ofboundary values.

[0171] Similarly, the line thickness may be made to change on the basisof a single boundary value in stead of making it change continuously ina manner as described above. Alternatively, the line thickness may bemade to change stepwise by providing a plurality of boundary values.

[0172] The technique of determining the angle of inclination of theendoscope 3 is not limited to the one described above.

[0173] It may be so arranged that the user can define the color and theline thickness that are used as attributes of the navigation-relatedinformation, the density of lines for drawing the model image, the sizeof the displayed pattern, the boundary values for changing the color andthe line thickness as a function of the distance and the characterstring of the characteristic information 28 indicating that theincapability of measurement of the apparatus.

[0174] The object of navigation may be a microscope.

[0175] Then, position of the focal point can be defined as object ofnavigation by obtaining the focal length of the microscope from themicroscope main body and replacing the coordinate of the front end ofthe endoscope 3 by that of the focal point.

[0176] While a navigation apparatus according to the invention isdescribed above in terms of the first and second embodiments, thepresent invention is by no means limited thereto and the embodiments canbe modified and/or alterered in various different ways without departingfrom the scope of the present invention.

[0177] With a navigation apparatus as set forth in claim 2 of theappended claims, a model image of the object or the target, informationon the direction of navigation and/or information on the distancebetween the object of navigation and the target are displayed wheneverthe position and orientation of the object in a three-dimensional spacecan be determined so that the user can easily comprehend the positionand orientation of the object of navigation in the three-dimensionalspace.

[0178] Additionally, when the apparatus is incapable of measuring theposition and orientation, it displays so and, therefore, the user caneasily be aware of the situation.

[0179] A navigation apparatus according to claim 2 covers both the abovedescribed first and second embodiments.

[0180] More specifically, while the above target may normally be apatient, a tumor to be surgically treated of a patient or an area of thebody of a patient requiring special attention during a surgicaloperation, it is by no means limited to an existing object ofexamination and may alternatively be a virtual target displayed as atwo-dimensional or three-dimensional image of a model synthesized byusing the video information of an existing target that is obtained inadvance.

[0181] While the above object may normally refer to an endoscope 3, itmay alternatively refer to some other surgical instrument such as asuction pipe or a pair of forceps.

[0182] While the above display section may normally refer to a liquidcrystal monitor, it may alternatively refer to some other videoinformation display such as a CRT display or a head mount display.

[0183] While the above model image refers to wireframe model data 10 inthe first embodiment, it may alternatively refer model data adapted toexpress a profile, including a popular data structure to be used forthree-dimensional computer graphics.

[0184] Additionally, while the above model image refers to thethree-dimensional volume data 11 of the target area in the first andsecond embodiments, it may alternatively take a form where a pluralityof two-dimensional pixel data exist.

[0185] While the above described information on the direction ofnavigation refers to the arrow 21 in the first embodiment, it mayalternatively be a two-dimensional geometric figure such as a triangleor a circle, a three-dimensional geometric figure such a cone or a setof visually recognizable image data.

[0186] While the above described distance information refers to thenumeral 31 indicating the distance to the tumor in the first embodiment,it may alternatively be a numeral indicating the distance to anappropriate target.

[0187] While it also refers to the bar 30 indicating the distance to thetumor in the first embodiment, it may alternatively be a two-dimensionalgeometric figure such as a triangle or a circle, a three-dimensionalgeometric figure such a cone or a set of visually recognizable imagedata.

[0188] While the information indicating an unmeasurable condition refersto the character information 28 of “unmeasurable condition” in the firstembodiment, it may alternatively include any character informationtelling the user that the apparatus is in an unmeasurable condition.

[0189] While it also refers to a yellow pixel frame 29 having a widthequal to 60 pixels in the first embodiment, it may alternatively referto any expression using symbols defined to indicate an unmeasurablecondition.

[0190] With a navigation apparatus as set forth in claim 3 of theappended claims, an image of an object acquired by the imaging sectionis displayed with other information on the object in an overlaid mannerso that the user can obtain an actual image of the object andnavigation-related information simultaneously and hence comprehend theposition, the profile and the condition of the object that he or shecannot see on the basis of the navigation-related information.

[0191] A navigation apparatus according to claim 3 covers both the abovedescribed first and second embodiments.

[0192] While the object refers to the endoscope 3 in the firstembodiment, it may alternatively refer to a microscope or some otherobject.

[0193] While the above display section normally refers to the liquidcrystal monitor in the first embodiment, it may alternatively refer tosome other video information display such as a CRT display or a headmount display.

[0194] With a navigation apparatus as set forth in claim 3 of theappended claims, the relative position and orientation of the target andthose of the object in a three-dimensional space are measured by meansof a three-dimensional position and orientation measuring section.

[0195] Then, the information generating section of the navigationapparatus generates information necessary for navigating the object onthe basis of the outcome of the measurement of the three-dimensionalposition and orientation measuring section.

[0196] Then, the display section of the navigation apparatus displaysnavigation-related information in a display mode selected out of aplurality of different display modes according to at least any ofdistance information on the distance between the object and the target,direction information on the direction of the target as viewed from theobject or information telling if the object or the target is foundwithin the effective area of measurement of the three-dimensionalposition and orientation measuring section or not.

[0197] As a result, the user can easily comprehend the distance betweenthe target and the object, the direction of the target as viewed fromthe object and if the object or the target is found within the effectivearea of measurement of the three-dimensional position and orientationmeasuring section or not.

[0198] A navigation apparatus according to claim 4 covers both the abovedescribed first and second embodiments.

[0199] Thus, while the target is a patient 1, a tumor to be surgicallytreated of a patient or an area of the body of a patient requiringspecial attention during a surgical operation in the above first andsecond embodiments, it is by no means limited to an existing object ofexamination and may alternatively be a virtual target displayed as atwo-dimensional or three-dimensional image of a model synthesized byusing the video information of an existing target that is obtained inadvance.

[0200] While the above object refers to an endoscope 3, it mayalternatively refer to some other surgical instrument such as a suctionpipe or a pair of forceps.

[0201] While the above three-dimensional position and orientationmeasuring section refers to sensors using LEDs for emitting infraredrays (sensing plates 2, 4, sensor assembly 7, sensor information storagesection 5 and sensor control section 6), it may alternatively refer to asensing system using magnetic sensors or a sensing system using a set ofmechanical links and joints, an encoder and a potentiometer popularlyused in ordinary three-dimensional position and orientation measuringsystems.

[0202] The information generating section refers to thenavigation-related information storage section 9 and thenavigation-related information control section 8.

[0203] While the display section normally refers to the liquid crystalmonitor 13, it may alternatively refer to some other video informationdisplay such as a CRT display or a head mount display.

[0204] For the purpose of the invention, the expression of “a pluralityof different display modes” refers to differences in color, in thethickness of line, in the dimensions of the drawing and in the densityof drawing lines.

[0205] With a navigation apparatus as set forth in claim 5 of theappended claims, a display section as set forth in claim 4 displays atleast profile information on the target or the object, internaltomographic information on the object, information on the direction ofthe object as viewed from the object or vice versa or information on thedistance to the object when the target or the object is measurable bythe three-dimensional position and orientation measuring section but itdisplays information telling that neither the target nor the object canbe measured.

[0206] A navigation apparatus according to claim 5 covers both the abovedescribed first and second embodiments.

[0207] More specifically, while the above profile information refers towireframe model data 10 in the first embodiment, it may also refer tomodel data adapted to express a profile, including a popular datastructure to be used for three-dimensional computer graphics.

[0208] Additionally, while it refers to the lines drawing the orthogonalprojection image 27 of the endoscope 3 in the second embodiment, it mayalso refer to expression techniques adapted to express a profile,including a popular data structure to be used for three-dimensionalcomputer graphics.

[0209] Still additionally, while the internal tomographic informationrefers to the three-dimensional volume data 11 of the target area in thefirst and second embodiments, it may alternatively take a form where aplurality of two-dimensional pixel data exist.

[0210] While the above described direction of the target refers to thearrow 21 in the first embodiment, it may alternatively be atwo-dimensional geometric figure such as a triangle or a circle, athree-dimensional geometric figure such a cone or a set of visuallyrecognizable image data.

[0211] While the above described distance information refers to thenumeral 31 indicating the distance to the tumor in the first embodiment,it may alternatively be a numeral indicating the distance to anappropriate target.

[0212] While it also refers to the bar 30 indicating the distance to thetumor in the first embodiment, it may alternatively be a two-dimensionalgeometric figure such as a triangle or a circle, a three-dimensionalgeometric figure such a cone or a set of visually recognizable imagedata.

[0213] While the information indicating an unmeasurable condition refersto the character information 28 of “unmeasurable condition” in the firstembodiment, it may alternatively include any character informationtelling the user that the apparatus is in an unmeasurable condition.

[0214] While it also refers to a yellow pixel frame 29 having a widthequal to 60 pixels in the first embodiment, it may alternatively referto any expression using symbols defined to indicate an unmeasurablecondition.

[0215] With a navigation apparatus as set forth in claim 6 of theappended claims, the object has an imaging section and the imageacquired by the imaging section is displayed with othernavigation-related information obtained by the information generatingsection in an overlaid manner.

[0216] Thus, the user can obtain an actual image of the object andnavigation-related information simultaneously and hence comprehend theposition, the profile and the condition of the object that he or shecannot see on the basis of the navigation-related information.

[0217] A navigation apparatus according to claim 6 covers the abovedescribed first embodiment.

[0218] While the object having an imaging section refers to theendoscope 3 in the first embodiment, it may alternatively refer to amicroscope or some other object.

[0219] While the above display section normally refers to the liquidcrystal monitor 13 in the first embodiment, it may alternatively referto some other video information display such as a CRT display or a headmount display.

[0220] With a navigation apparatus as set forth in claim 7 of theappended claims, the navigation-related information displayed on thedisplay section changes its color as a function of the relative distancebetween the target and the object as measured by the three-dimensionalposition and orientation measuring section so that the user can visuallycomprehend with ease a situation where the relative distance is made toosmall.

[0221] Alternatively, it may be so arranged that both the relativedistance and the direction toward the target as viewed from the objectare evaluated at the same time and the color of the displayedinformation is changed depending on the situation. Then, the user canvisually comprehend both the relative distance and the direction withease.

[0222] A navigation apparatus according to claim 7 covers both the abovedescribed first and second embodiments.

[0223] More specifically, while the color of the displayednavigation-related information refers to that of the wireframe image 18of the target area and the internal tomographic image 19 displayed onthe monitor in the first embodiment, it may also refer to the color ofthe arrow 21 of the first embodiment.

[0224] It may additionally refers to the color of the tri-sectionalimage 26 of the target area and the orthogonal projection image 27 ofthe endoscope 3 in the second embodiment.

[0225] With a navigation apparatus as set forth in claim 8 of theappended claims, the thickness of the lines of the navigation-relatedinformation displayed on the display section changes as a function ofthe relative distance between the target and the object as measured bythe three-dimensional position and orientation measuring section so thatthe user can visually comprehend with ease a situation where therelative distance is made too small.

[0226] Additionally, as the thickness of the lines of thenavigation-related information displayed on the display section changesas a function of the direction of the target as viewed from the object,the user can visually comprehend with ease a situation where therelative direction is deviating from the right direction.

[0227] Alternatively, it may be so arranged that both the relativedistance and the direction toward the target as viewed from the objectare evaluated at the same time and the line thickness of the displayedinformation is changed depending on the situation. Then, the user canvisually comprehend both the relative distance and the direction withease.

[0228] A navigation apparatus according to claim 8 covers both the abovedescribed first and second embodiments.

[0229] More specifically, while the line thickness of the displayednavigation-related information refers to that of the wireframe image 18of the target area and the internal tomographic image 19 displayed onthe monitor in the first embodiment, it may also refer to the linethickness of the arrow 21 of the first embodiment.

[0230] It may additionally refers to line thickness of the tri-sectionalimage 26 of the target area and the orthogonal projection image 27 ofthe endoscope 3 in the second embodiment.

[0231] With a navigation apparatus as set forth in claim 9 of theappended claims, the profile model of the target and the internaltomographic image are switched from one to the other on the displaysection as a function of the relative distance between the target andthe object as measured by the three-dimensional position and orientationmeasuring section so that the user can visually comprehend with easethat the object is located close to the target.

[0232] Thus, the user can get necessary information with ease dependingon if the distance between the object and the target is smaller than apredetermined value or not.

[0233] A navigation apparatus according to claim 9 covers the abovedescribed first embodiment.

[0234] More specifically, while the above profile model refers towireframe image 18 in the first embodiment, it may also refer to variousprofiles used in three-dimensional computer graphics such as polygon aswell as contour lines and equidistant curves drawn relative to theviewing direction. The above internal tomographic image refers to theinternal tomographic image 19 of the first embodiment.

[0235] With a navigation apparatus as set forth in claim 10 of theappended claims, the density of lines drawing the target model image isfinely lowered when the relative distance between the target and theobject is large and finely raised when the relative distance is small soas to make the load of drawing the target image and the quantity ofinformation used for displaying the image may be well balanced.

[0236] As a result, the user can obtain an adequate amount ofinformation that is displayed with an adequate drawing rate depending asa function of the relative distance between the target and the object.

[0237] A navigation apparatus according to claim 10 covers the abovedescribed first embodiment.

[0238] More specifically, while the above profile model refers towireframe image 18 in the first embodiment, it may also refer to variousprofiles used in three-dimensional computer graphics such as polygon aswell as contour lines and equidistant curves drawn relative to theviewing direction.

[0239] With a navigation apparatus as set forth in claim 11 of theappended claims, the information generating section computationallydetermines the positional relationship of the image to be displayed andthe display area of the display section on the basis of the relativedistance between the target and the object and the relative direction ofthe target as viewed from the object and simply indicates the directionof the target when no image is displayed in the display area for thetarget so that the user can comprehend the relative positions of thetarget and the object and the direction of the target as viewed from theobject without missing either of them by selecting the display of thedirection when no image is display for the target.

[0240] A navigation apparatus according to claim 11 covers the abovedescribed first embodiment.

[0241] More specifically, while the above profile model refers towireframe image 18 in the first embodiment, it may also refer to variousprofiles used in three-dimensional computer graphics such as polygon aswell as contour lines and equidistant curves drawn relative to theviewing direction along with information on the internal tomographicimage.

[0242] While the above described information on the direction ofnavigation refers to the arrow 21 in the first embodiment, it mayalternatively be a two-dimensional geometric figure such as a triangleor a circle, a three-dimensional geometric figure such a cone or a setof visually recognizable image data.

[0243] With a navigation apparatus as set forth in claim 12 of theappended claims, the relative distance between the target and the objectis indicated by the size or the shape of a symbol so that the user canvisually comprehend with ease not only the distance but also thedirection of the target as viewed from the object.

[0244] A navigation apparatus according to claim 12 covers the abovedescribed first embodiment.

[0245] More specifically, while the above described symbol refers to thearrow 21 in the first embodiment, it may alternatively be atwo-dimensional geometric figure such as a triangle or a circle, athree-dimensional geometric figure such a cone or a set of visuallyrecognizable image data.

[0246] With a navigation apparatus as set forth in claim 13 of theappended claims, the relative distance between the target and the objectof navigation and their orientations in a three-dimensional space aredetermined by the three-dimensional position and orientation measuringsection.

[0247] Then, an computational information determining section generatesnavigation-related information such as information three-dimensionalposition and orientation information on the target and the object ofnavigation including the relative distance between the target and theobject of navigation and their orientations and if they are measurableor not and controls the generated information.

[0248] Then, the information display section displays thenavigation-related information generated by the computationalinformation determining section.

[0249] As a result, the user can easily comprehend the positionalrelationship between the target and the object of navigation includingtheir orientations and if they are measurable or not.

[0250] A navigation apparatus according to claim 13 covers both thefirst and second embodiments.

[0251] More specifically, while the above target may normally be apatient, a tumor to be surgically treated of a patient or an area of thebody of a patient requiring special attention during a surgicaloperation, it is by no means limited to an existing object ofexamination and may alternatively be a virtual target displayed as atwo-dimensional or three-dimensional image of a model synthesized byusing the video information of an existing target that is obtained inadvance.

[0252] While the above object may normally refer to an endoscope 3, itmay alternatively refer to some other surgical instrument such as asuction pipe or a pair of forceps.

[0253] While the above three-dimensional position and orientationmeasuring section refers to sensors using LEDs for emitting infraredrays (sensing plates 2, 4, sensor assembly 7, sensor information storagesection 5 and sensor control section 6), it may alternatively refer to asensing system using magnetic sensors or a sensing system using a set ofmechanical links and joints, an encoder and a potentiometer popularlyused in ordinary three-dimensional position and orientation measuringsystems.

[0254] The computational information determining section refers to thenavigation-related information storage section 9 and thenavigation-related information control section 8.

[0255] While the information display section refers to the liquidcrystal monitor 13, it may alternatively refer to some other videoinformation display such as a CRT display or a head mount display.

[0256] The expression “attributes of navigation-related information” asused herein refers to the color, the thickness of line, the dimensionsthe drawing and the density of drawing lines.

[0257] With a navigation apparatus as set forth in claim 14 of theappended claims, the navigation-related information includes a modelimage of the profile of the target and/or that of the object ofnavigation, a model image of the internal tomographic information of thetarget, a symbol pattern indicating the direction in which the targetand/or the object of navigation will be found and/or a numerical value oa symbol pattern indicating the distance between the target and theobject of navigation when the three-dimensional position and orientationmeasuring section is operating normally.

[0258] When the three-dimensional position and orientation measuringsection is inoperative, the navigation-related information refers tocharacter information or a symbol pattern indicating that thethree-dimensional position and orientation measuring section is inoperative.

[0259] A navigation apparatus according to claim 14 covers both thefirst and second embodiments.

[0260] More specifically, while the above model image of the profilerefers to wireframe model data 10 in the first embodiment, it may alsorefer to model data adapted to express a profile, including a populardata structure to be used for three-dimensional computer graphics.

[0261] Additionally, while it refers to the lines drawing the orthogonalprojection image 27 of the endoscope 3 in the second embodiment, it mayalso refer to expression techniques adapted to express a profile,including a popular data structure to be used for three-dimensionalcomputer graphics.

[0262] Still additionally, while the model image of the internaltomographic information refers to the three-dimensional volume data 11of the target area in the first and second embodiments, it mayalternatively take a form where a plurality of two-dimensional pixeldata exist.

[0263] While the above described symbol pattern indicating the directionin which the object of navigation will be found refers to the arrow 21in the first embodiment, it may alternatively be a two-dimensionalgeometric figure such as a triangle or a circle, a three-dimensionalgeometric figure such a cone or a set of visually recognizable imagedata.

[0264] While the above described numerical value indicating the distanceto the target refers to the numeral 31 indicating the distance to thetumor in the first embodiment, it may alternatively be a numeralindicating the distance to an appropriate target.

[0265] While it also refers to the bar 30 indicating the distance to thetumor in the first embodiment, it may alternatively be a two-dimensionalgeometric figure such as a triangle or a circle, a three-dimensionalgeometric figure such a cone or a set of visually recognizable imagedata.

[0266] While the character information indicating an unmeasurablecondition refers to the character information 28 of “unmeasurablecondition” in the first embodiment, it may alternatively include anycharacter information telling the user that the apparatus is in anunmeasurable condition.

[0267] While it also refers to a yellow pixel frame 29 having a widthequal to 60 pixels in the first embodiment, it may alternatively referto any expression using symbols defined to indicate an unmeasurablecondition.

[0268] With a navigation apparatus as set forth in claim 15 of theappended claims, the object of navigation has an observational functionand the observed image obtained by means of the observational functionis displayed with other navigation-related information obtained by thecomputational information determining section in an overlaid manner.

[0269] Thus, the user can obtain an actual image of the object andnavigation-related information simultaneously and hence comprehend theposition, the profile and the condition of the object that he or shecannot see on the basis of the navigation-related information.

[0270] A navigation apparatus according to claim 15 covers the abovedescribed first embodiment.

[0271] While the object of navigation having an observational functionrefers to the endoscope 3 in the first embodiment, it may alternativelyrefer to a microscope or some other object.

[0272] While the above information display section normally refers tothe liquid crystal monitor 13 in the first embodiment, it mayalternatively refer to some other video information display such as aCRT display or a head mount display.

[0273] With a navigation apparatus as set forth in claim 16 of theappended claims, the navigation-related information displayed on thedisplay section changes its color as a function of the relative distancebetween the target and the object as measured by the three-dimensionalposition and orientation measuring section so that the user can visuallycomprehend with ease a situation where the relative distance is made toosmall.

[0274] Alternatively, it may be so arranged that the navigation-relatedinformation displayed on the display section changes its color as afunction of the relative direction of the target and the object ofnavigation so that the user also can visually comprehend a situationwhere the relative direction is deviated from the right direction.

[0275] Still alternatively, the color may be made to changesimultaneously as a function of both the relative distance and therelative direction. Then, the user can visually comprehend both therelative distance and the direction with ease.

[0276] A navigation apparatus according to claim 16 covers both theabove described first and second embodiments.

[0277] More specifically, while the color of the displayednavigation-related information refers to that of the wireframe image 18of the target area and the internal tomographic image 19 displayed onthe monitor in the first embodiment, it may also refer to the color ofthe arrow 21 of the first embodiment.

[0278] It may additionally refers to the color of the tri-sectionalimage 26 of the target area and the orthogonal projection image 27 ofthe endoscope 3 in the second embodiment.

[0279] With a navigation apparatus as set forth in claim 17 of theappended claims, the thickness of the lines of the navigation-relatedinformation obtained by the three-dimensional position and orientationmeasuring section is made to vary as a function of the relative distancebetween the target and the object of navigation so that the user canvisually comprehend with ease a situation where the relative distancehas become too small.

[0280] Alternatively, it may be so arranged that the thickness of thelines of the navigation-related information vary as a function of thedirection to the target as viewed from the object of navigation so thatthe user can also visually comprehend with ease a situation where therelative direction has deviated.

[0281] A navigation according to claim 17 covers both the first andsecond embodiments.

[0282] While the thickness of the lines of navigation-relatedinformation refers to the wireframe image 18 of the target area drawn onthe monitor in the first embodiment, it may also include the arrow 21 inthe first embodiment.

[0283] It refers to the orthogonal projection image 27 of the endoscope3 drawn on the monitor in the second embodiment.

[0284] With a navigation apparatus as set forth in claim 18 of theappended claims, the profile model of the target and the internaltomographic image are switched from one to the other on the displaysection as a function of the relative distance between the target andthe object as measured by the three-dimensional position and orientationmeasuring section so that the user can visually comprehend with easethat the object is located close to the target.

[0285] Thus, the user can get necessary information with ease dependingon if the distance between the object and the target is smaller than apredetermined value or not.

[0286] A navigation apparatus according to claim 18 covers the abovedescribed first embodiment.

[0287] More specifically, while the above profile model refers towireframe image 18 in the first embodiment, it may also refer to variousprofiles used in three-dimensional computer graphics such as polygon aswell as contour lines and equidistant curves drawn relative to theviewing direction.

[0288] The above internal tomographic image refers to the internaltomographic image 19 of the first embodiment.

[0289] With a navigation apparatus as set forth in claim 19 of theappended claims, the density of lines drawing the target model image isfinely lowered when the relative distance between the target and theobject is large and finely raised when the relative distance is small soas to make the load of drawing the target image and the quantity ofinformation used for displaying the image may be well balanced.

[0290] As a result, the user can obtain an adequate amount ofinformation that is displayed with an adequate drawing rate depending asa function of the relative distance between the target and the object.

[0291] A navigation apparatus according to claim 19 covers the abovedescribed first embodiment.

[0292] More specifically, while the above profile model refers towireframe image 18 in the first embodiment, it may also refer to variousprofiles used in three-dimensional computer graphics such as polygon aswell as contour lines and equidistant curves drawn relative to theviewing direction.

[0293] With a navigation apparatus as set forth in claim 20 of theappended claims, the computational information determining sectioncomputationally determines the positional relationship of the image tobe displayed and the display area of the display section on the basis ofthe relative distance between the target and the object of navigationand the relative direction of the target as viewed from the object andonly a symbol pattern is displayed when no model image is displayed inthe display area so that the user can comprehend the relative positionsof the target and the object and the direction of the target as viewedfrom the object without missing either of them by selecting the displayof the direction when no image is display for the target.

[0294] A navigation apparatus according to claim 20 covers the abovedescribed first embodiment.

[0295] More specifically, while the above profile model refers towireframe image 18 in the first embodiment, it may also refer to variousprofiles used in three-dimensional computer graphics such as polygon aswell as contour lines and equidistant curves drawn relative to theviewing direction along with information on the internal tomographicimage.

[0296] While the above described symbol pattern refers to the arrow 21in the first embodiment, it may alternatively be a two-dimensionalgeometric figure such as a triangle or a circle, a three-dimensionalgeometric figure such a cone or a set of visually recognizable imagedata.

[0297] With a navigation apparatus as set forth in claim 21 of theappended claims, the relative distance between the target and the objectis indicated by the size of a pattern so that the user can visuallycomprehend with ease not only the distance but also the direction of thetarget as viewed from the object.

[0298] A navigation apparatus according to claim 21 covers the abovedescribed first embodiment.

[0299] More specifically, while the above described symbol patternrefers to the arrow 21 in the first embodiment, it may alternatively bea two-dimensional geometric figure such as a triangle or a circle, athree-dimensional geometric figure such a cone or a set of visuallyrecognizable image data.

[0300] As described above, both the above described first and secondembodiments of navigation apparatus according to the invention areadapted to modify the navigation-related information displayed on thedisplay section as a function of the relative three-dimensionalpositions and orientations of the target and the object of navigation tomake the user easily comprehend the distance between the target and theobject and obtain navigation-related information of necessary type withease.

[0301] Now, the third and fourth embodiments of the invention will bedescribed. They are surgical operation image acquisition/displayapparatus realized by applying a navigation apparatus according to theinvention.

[0302] (Embodiment 3)

[0303]FIG. 14 is a schematic block diagram of the third embodiment ofthe invention which is a surgical operation image acquisition/displayapparatus, showing its configuration; The third embodiment of surgicaloperation image acquisition/display apparatus according to the inventionhas a configuration as described below.

[0304] Referring to FIG. 14, the surgical operation imageacquisition/display apparatus comprises a surgical microscope 141 (firstobservation section) and an endoscope 142 (second observation section)for observing an area located in the dead angle of the surgicalmicroscope 141.

[0305] The surgical microscope 141 includes an microscope optical system141 a mounted on a stand (not shown), a microscope camera 141 b attachedto the microscope optical system 141 a and a microscope camera controlunit (hereinafter referred to as microscope CCU) 141 c for convertingthe output of the microscope camera 141 b into a video signal.

[0306] The microscope optical system 141 a is provided with illuminationlight emitted from a light source (not shown) and guided by a lightguide (not shown) for the purpose of observation.

[0307] On the other hand, the endoscope 142 includes an endoscopeoptical system 142 a, an endoscope camera 142 b attached to theendoscope optical system 142 a and an endoscope camera control unit(hereinafter referred to as endoscope CCU) 142 for converting the outputof the endoscope camera into a video signal.

[0308] The endoscope optical system 142 a is provided with illuminationlight emitted from a light source (not shown) and guided by a lightguide (not shown) for the purpose of observation.

[0309] The video output of the microscope CCU 141 c and that of theendoscope CCU 142 c are fed to a video mixer 143 (observed imagesynthesizing section).

[0310] As illustrated in FIGS. 15A through 15F, the video mixer 143 hasa plurality of display modes 0 through 5.

[0311] In the display mode 0 (FIG. 15A), the video output of themicroscope CCU 141 c is displayed without being processed.

[0312] In the display mode 1 (FIG. 15B), the video output of theendoscope CCU 142 c is displayed without being processed.

[0313] In the display mode 2 (FIG. 15C), the video output of theendoscope CCU 142 c is dimensionally reduced and displayed in the videooutput of the microscope CCU 141 c.

[0314] In the display mode 3 (FIG. 15D), the video output of themicroscope CCU 141 c is dimensionally reduced and displayed in the videooutput of the endoscope CCU 142 c.

[0315] The extent of dimensional reduction and the display position ofthe dimensionally reduced output are variable both in the display mode 2and the display mode 3.

[0316] In the display mode 4 (FIG. 15E), which is a variation of thedisplay mode 2 and in which the video output of the endoscope CCU 142 cis dimensionally reduced and displayed at the right top corner of thevideo output of the microscope CCU 141 c, the video output of theendoscope CCU 142 c is dimensionally reduced to the same extent anddisplayed at the left bottom corner of the video output of themicroscope CCU 141 c.

[0317] Similarly, in the display mode 5 (FIG. 15F), which is a variationof the display mode 3 and in which the video output of the microscopeCCU 141 c is dimensionally reduced and displayed at the right top cornerof the video output of the endoscope CCU 142 c, the video output of themicroscope CCU 141 c is dimensionally reduced to the extent smaller thanthat of FIG. 15D and displayed at the right top corner of the videooutput of the endoscope CCU 142 c.

[0318] The mode of display and the position and the size of thedisplayed image (or each of the displayed images) as defined by thevideo mixer 143 can be appropriately changed by means of an externallyapplied control signal. More specifically, the displayed image(s) can beenlarged or reduced independently with an appropriately selectedmagnification factor.

[0319] Then, the output of the video mixer 143 is fed to a liquidcrystal display 144 (display section) and displayed to the surgicaloperator for observation.

[0320] The combination of the video mixer 143 and the liquid crystaldisplay 144 corresponds to the image display section as used in theappended claims.

[0321] Referring to FIG. 14, position and orientation sensor 145(position and orientation detection section) comprises hard sensingplates 145 b and 145 c, each having three infrared light emitting diodes(LEDs) 145 a for emitting arranged at the respective corners of atriangle, a sensor assembly 145 d for detecting the quantity of lightemitted from each of the infrared LEDs 145 a for emitting and a sensorcontroller 145 e for computationally determining the three-dimensionalposition and orientation of the sensing plate 145 b and that of thesensing plate 145 c from the output of the sensor assembly.

[0322] Note that the position of each of the LEDs 145 a for emittinginfrared is observed and determined in advance in terms of thecoordinate system defined on each of the sensing plates 145 b and 145 cand stored in the sensor controller 145 e as LED definition data.

[0323] Assume that the sensing plate 145 b is attached to the head ofthe patient 146 in such a way that its position and orientation relativeto the head would not change easily.

[0324] The other sensing plate 145 c is attached to the endoscope 143 bya mount section (not shown).

[0325] The sensor controller 145 e is connected to an image controller147 (state of synthesis modification specifying section).

[0326] The image controller 147 is connected to the video mixer 143.

[0327] Assume that the area of operation to be surgically treated andcertain characteristic points on the patient are observed by means of CTor MRI and the three-dimensional positions thereof are computationallydetermined by an image processing computer (not shown) and stored in theimage controller 147 as data on the area and the characteristic points.

[0328] At this time, as shown in FIG. 16, the data on the area ofsurgical operation and the patient 146 are correlated by observing thecoordinate values of the characteristic points in the model datacoordinate system m and those of the characteristic points on thepatient 146 in the patient coordinate system p defined by the sensingplate 145 b and computing a coordinate transformation matrix pHm.

[0329] The coordinate transformation matrix pHm is stored in the storagesection of the image controller 147.

[0330] Similarly, the coordinate values of the front end and those ofthe rear end of the endoscope 142 are observed in terms of the endoscopecoordinate system e defined by the sensing plate 145 c attached to theendoscope 142 and stored in the storage section of the image controller147.

[0331] Now, the operation of the third embodiment will be describedbelow.

[0332] When the surgical operation image acquisition/display apparatusis used, the surgical microscope 141 and the endoscope 142 are combinedfor use as shown in FIG. 14.

[0333] When the surgical operation image acquisition/display apparatusis in operation, the sensor controller 145 e drives the infrared LEDs145 a to sequentially emit and determines the three-dimensional positionof each of the infrared LEDs 145 a on the basis of its output. At thesame time, the sensor controller 145 e computationally determines thethree-dimensional position and orientation of the sensing plate 145 band that of the sensing plate 145 c, using the LED definition datastored in the sensor controller 145 e and outputs the obtained data tothe image controller 147 upon request.

[0334] The image controller 147 computes the coordinate transformationmatrix pHe from the sensing plate 145 b of the patient coordinate systemp attached to the head of the patient 146 to the sensing plate 145 c ofthe endoscope coordinate system e attached to the endoscope 142 on thebasis of the three-dimensional position and orientation information.

[0335] The image controller 147 also computationally determines therelative distance between the patient 146 and the endoscope 142 and therelative direction of the patient 146 as viewed from the endoscope 142on the basis of the coordinate transformation matrix pHe and the abovedescribed coordinate transformation matrix pHm.

[0336] The processing operation of the image controller 147 will bedescribed further by referring to the flow chart of FIG. 18.

[0337] Firstly, the image controller 147 outputs a request signalcyclically with a predetermined period (e.g., 33 msec) to the sensorcontroller 145 e to receive the three-dimensional position andorientation information on the sensing plates 145 b and 145 c from thesensor controller 145 e (Step S101).

[0338] Then, the image controller 147 judges if the endoscope 142 islocated close to the area of surgical operation (e.g., within a range of50 mm) on the basis of the received three-dimensional position andorientation information (Step S102).

[0339] If the image controller 147 judges that the endoscope 142 islocated close to the area of surgical operation, it outputs aninstruction for switching from the microscope image to the endoscopeimage (mode 1) to the video mixer 143 (Step S103).

[0340] If, on the other hand, the image controller 147 judges that theendoscope 142 is not located close to the area of surgical operation, itoutputs an instruction for switching from the endoscope image to themicroscope image (mode 0) to the video mixer 143 (Step S104).

[0341] When, the endoscope 142 is moved away from the area of surgicaloperation and the area is observed mainly by the surgical microscope 141as a result of the above judgment, the image observed through thesurgical microscope 141 is displayed on the liquid crystal display 144.On the other hand, when the endoscope 142 is moved closer to the area ofsurgical operation, the image observed through the endoscope 142 isdisplayed on the liquid crystal display 144.

[0342] Because the image observed through the surgical microscope isdisplayed on the liquid crystal display 144 when the endoscope 142 ismoved away from the area of surgical operation, the surgeon can see thedesired image without paying effort for switching from the microscopeimage to the endoscope image.

[0343] It may be needless to say that the configuration of thisembodiment can be modified and/or altered in various different wayswithout departing from the scope of the invention.

[0344] For instance, while the first observation section refers to thesurgical microscope 141 in this embodiment, it may alternatively be theendoscope 142 or some other observation sections or units.

[0345] While the second observation section refers to the endoscope 142in this embodiment, it may alternatively be the surgical microscope 141or some other observation sections or units.

[0346] While the image synthesizing section refers to the video mixer143 in this embodiment, it may be some other section for externallymodifying the synthesized state of a plurality of images.

[0347] While the display section refers to the liquid crystal display144 in this embodiment, it may alternatively be a CRT display, a headmounted display or a projector adapted to display video signals.

[0348] While the position and orientation detection section is theposition and orientation sensor (comprising the infrared LEDs 145 a foremitting, the sensing plates 145 b, 145 c, the sensor assembly 145 d andthe sensor controller 145 e) in this embodiment, it may alternatively beany appropriate section for detecting the three-dimensional position andorientation of an object such as a magnetic sensor or a set ofmechanical links and joints, encoders and potentiometers.

[0349] While the state of synthesis modification specifying section ofthis embodiment uses the distance between the area of surgical operationand the endoscope 142 for its judgment, it may alternatively use boththe distance and the direction of the area as viewed from the endoscope142.

[0350] While the position and orientation detection section of thisembodiment detects the position and orientation of the endoscope 142, itmay alternatively detect the position and orientation of the microscope141 or both the position and orientation of the endoscope 142 and thatof the microscope 141.

[0351] (Embodiment 4)

[0352] Now, the fourth embodiment of the invention, which is also asurgical operation image acquisition/display apparatus, will bedescribed below.

[0353] The fourth embodiment of surgical operation imageacquisition/display apparatus has a configuration substantially same asthe above described third embodiment and hence the similar components inthe graphic illustrations thereof will be denoted respectively by thesame reference symbols and will not be described any further.

[0354] Additionally, since the operational function of the fourthembodiment of surgical operation image acquisition/display apparatus issame as that of the above described third embodiment except theprocessing operation of the image controller 147, only the latter willbe discussed hereinafter.

[0355] More specifically, in this embodiment, one of the obtained twoimages is dimensionally reduced and synthetically combined with theother image so that they may be displayed simultaneously on the displayscreen for observation.

[0356] The processing operation of the image controller 147 will bediscussed below by referring to the flow chart of FIG. 19.

[0357] Firstly, the image controller 147 outputs a request signalcyclically with a predetermined period (e.g., 33 msec) to the sensorcontroller 145 e to receive the three-dimensional position andorientation information on the sensing plates 145 b and 145 c from thesensor controller 145 e (Step S201).

[0358] Then, the image controller 147 judges if the endoscope 142 islocated close to the area of surgical operation (e.g., within a range of50 mm) on the basis of the received three-dimensional position andorientation information (Step S202).

[0359] If the image controller 147 judges that the endoscope 142 islocated close to the area of surgical operation, it outputs aninstruction for dimensionally reducing the microscope image and displayit with the endoscope image (mode 3) to the endoscope image (mode 1) tothe video mixer 143 (Step S203).

[0360] If, on the other hand, the image controller 147 judges that theendoscope 142 is not located close to the sit of surgical operation, itoutputs an instruction for dimensionally reducing the endoscope imageand display it with the microscope image (mode 2) to the video mixer 143(Step S204).

[0361] When, the endoscope 142 is moved away from the area of surgicaloperation and the area is observed mainly by the surgical microscope 141as a result of the above judgment, the dimensionally reduced imageobserved through the endoscope 142 is displayed with the image observedthrough the surgical microscope 141 on the liquid crystal display 144.

[0362] On the other hand, when the endoscope 142 is moved closer to thearea of surgical operation, the dimensionally reduced image observedthrough the surgical microscope 141 is displayed with the image observedthrough the endoscope 142 on the liquid crystal display 144.

[0363] Because the dimensionally reduced image observed through theendoscope 142 is displayed with the image observed through the surgicalmicroscope on the liquid crystal display 144 when the endoscope 142 ismoved away from the area of surgical operation, the surgeon can see thedesired plurality of images without paying effort for switching from themicroscope image to the endoscope image.

[0364] It may be needless to say that the configuration of this fourthembodiment can be modified and/or altered in various different wayswithout departing from the scope of the invention.

[0365] For instance, while the first observation section refers to thesurgical microscope 141 in these embodiments, it may alternatively bethe endoscope 142 or some other observation sections or units.

[0366] While the second observation section refers to the endoscope 142in these embodiments, it may alternatively be the surgical microscope141 or some other observation sections or units.

[0367] While the image synthesizing section refers to the video mixer143 in these embodiments, it is by no means limited thereto and may besome other section for externally modifying the synthesized state of aplurality of images.

[0368] While the display section refers to the liquid crystal display144 in these embodiments, it may alternatively be a CRT display, a headmounted display or a projector adapted to display video signals.

[0369] While the position and orientation detection section is theposition and orientation sensor (comprising the infrared LEDs 145 a foremitting, the sensing plates 145 b, 145 c, the sensor assembly 145 d andthe sensor controller 145 e) in these embodiments, it may alternativelybe any appropriate section for detecting the three-dimensional positionand orientation of an object such as a magnetic sensor or a set ofmechanical links and joints, encoders and potentiometers.

[0370] The state of synthesis modification specifying section refers tothe image controller in these embodiments.

[0371] As described above, both the third and fourth embodiments of theinvention provide a surgical operation image acquisition/displayapparatus that efficiently assists a surgeon to smoothly carry out asurgical operation without requiring him or her to switch theobservation system from one to another by means of a navigationapparatus when the surgical operation is conducted by using a pluralityof observation systems including a surgical microscope and an endoscope.

[0372] Additional advantages and modifications will readily occur tothose skilled in the art. Therefore, the invention in its broaderaspects is not limited to the specific details and representativeembodiments shown and described herein. Accordingly, variousmodifications may be made without departing from the spirit or scope ofthe general inventive concept as defined by the appended claims andtheir equivalents.

What is claimed is:
 1. A navigation apparatus comprising: anavigation-related information generating section for generatingnavigation-related information by measuring the relative position andorientation of an object and a target in a three-dimensional space inorder to navigate said object to said target; and a display section fordisplaying said navigation-related information generated by saidnavigation-related information generating section in different modesaccording to the relative position and orientation of said object andsaid target.
 2. A navigation apparatus comprising: a navigation-relatedinformation generating section for generating navigation-relatedinformation by measuring the relative position and orientation of anobject and a target in a three-dimensional space in order to navigatesaid object to said target; and a display section for displaying atleast a model image of the object or the target, information on thedirection of navigation or information on the distance between theobject of navigation and the target when said navigation-relatedinformation generating section can measure said position and orientationin a three-dimensional space but displaying information indicating ameasurement incapable situation when said navigation-related informationgenerating section cannot measure said position and orientation.
 3. Anavigation apparatus according to claims 1 and 2, wherein said objecthas an image acquisition function; and said display section can displaythe image acquired by the image acquisition function of said object withother navigation-related information in an overlaid.
 4. A navigationapparatus adapted to generate information for navigating an object to atarget comprising: a three-dimensional position and orientationmeasuring section for measuring the three-dimensional position andorientation of said object and that of said target; an informationgenerating section for generating information necessary for thenavigation on the basis of the obtained result of said three-dimensionalposition and orientation measuring section; and a display section fordisplaying the information generated by said information generatingsection; wherein said display section displays navigation-relatedinformation according to at least information on the distance betweensaid object and said target, information on the direction toward saidtarget as viewed from said object and information if either said objector said target is located within the effective area of measurement ofsaid three-dimensional position and orientation measuring section in adisplay mode selected out of a plurality of display modes.
 5. Anavigation apparatus according to claim 4 , wherein said display sectiondisplays at least profile information on said target or said object,internal tomographic information on said target, information on thedirection toward said target as viewed from said object or vice versa orinformation on the distance between said object and said target whensaid target and said object is measurable by said three-dimensionalposition and orientation measuring section but displays informationindicating a measurement incapable situation when neither said targetnor said object is measurable by said three-dimensional position andorientation measuring section.
 6. A navigation apparatus according toclaim 4 , wherein said object has an image acquisition function; andsaid display section can display the image acquired by the imageacquisition function of said object with other navigation-relatedinformation in an overlaid.
 7. A navigation apparatus according to claim4 , wherein said navigation-related information displayed on the displaysection changes its color as a function of the relative distance betweensaid target and said object or the direction toward said target asviewed from said object.
 8. A navigation apparatus according to claim 4, wherein the thickness of the lines of the navigation-relatedinformation displayed on the display section changes as a function ofthe relative distance between said target and said object or thedirection toward said target as viewed from said object.
 9. A navigationapparatus according to claim 4 , wherein a profile model of the targetand an internal tomographic image are displayed alternatively on saiddisplay section as a function of the distance between said target andsaid object.
 10. A navigation apparatus according to claim 4 , whereinthe density of lines displayed on said display section for drawing saidtarget model image is varied as a function of the distance between saidtarget and said object.
 11. A navigation apparatus according to claim 4, wherein an image representing said target and directional informationof said target are displayed alternatively on said display section as afunction of the distance between said target and said object or thedirection toward said target as viewed from said object.
 12. Anavigation apparatus according to claim 4 , wherein directioninformation of said target is displayed on said display section in theform of a symbol at least whose size or shape varies as a function ofthe distance between said target and said object or the direction towardsaid target as viewed from said object.
 13. A navigation apparatuscomprising: a target of navigation; an object to be navigated to saidtarget; a three-dimensional position and orientation measuring sectionfor measuring the three-dimensional position and orientation of at leastsaid target or said object of navigation; a computational informationdetermining section for generating navigation-related information on thebasis of the information obtained by the measurement of saidthree-dimensional position and orientation measuring section andcontrolling the generated information; and an information displaysection for display the navigation-related information generated andcontrolled by said computational information determining section; therelative distance between the target and the object of navigation andtheir orientations in a three-dimensional space are determined by thethree-dimensional position and orientation measuring section; whereinsaid computational information determining section modifies either theattribute or the type of said navigation-related information as afunction of at least the relative distance between said target and saidobject of navigation or the direction toward said target as viewed fromsaid object measured by said three-dimensional position and orientationmeasuring section or the immeasurability thereof so as to make itvisibly reflect the outcome of measurement.
 14. A navigation apparatusaccording to claim 13 , wherein said navigation-related informationincludes a model image of the profile of the target and/or that of theobject of navigation, a model image of the internal tomographicinformation of the target, a symbol pattern indicating the direction inwhich the target and/or the object of navigation will be found and/or anumerical value or a symbol pattern indicating the distance between thetarget and the object of navigation when said three-dimensional positionand orientation measuring section is operating normally whereas itincludes character information or a symbol pattern indicating animmeasurable situation when the three-dimensional position andorientation measuring section is inoperative.
 15. A navigation apparatusaccording to claim 13 , wherein said object of navigation has an imageacquisition function and the acquired image obtained by means of theimage acquisition function is displayed with other navigation-relatedinformation obtained by said computational information determiningsection in an overlaid manner by means of said computational informationdetermining section.
 16. A navigation apparatus according to claim 13 ,wherein the navigation-related information displayed on the displaysection changes its color as a function of the relative distance betweensaid target and said object of navigation or the direction toward saidtarget as viewed from said object of navigation.
 17. A navigationapparatus according to claim 13 , wherein the thickness of the lines ofthe navigation-related information displayed on the display sectionchanges as a function of the relative distance between said target andsaid object of navigation and/or the direction toward the target asviewed from the object of navigation.
 18. A navigation apparatusaccording to claim 13 , wherein the profile model of said target and theinternal tomographic image are switched from one to the other on thedisplay section as a function of the relative distance between saidtarget and said object of navigation.
 19. A navigation apparatusaccording to claim 13 , wherein the density of lines of the displayedtarget model image is modified as a function of the relative distancebetween said target and said object of navigation.
 20. A navigationapparatus according to claim 13 , wherein a model image of at leasteither said target or said object of navigation or a symbol patternindicating the direction in which at least said target or said object ofnavigation is found is displayed alternatively as a function of therelative distance between said target and said object of navigation andthe direction toward the target as viewed from the object of navigation.21. A navigation apparatus according to claim 13 , wherein the size ofthe symbol pattern indicating the direction of said target as viewedfrom the object of navigation is modified as a function of the relativedistance between said target and said object of navigation.
 22. Asurgical operation image acquisition/display apparatus comprising: anobservation section having a plurality of observation units and adaptedto modify its position and orientation; an image display section adaptedto alternatively or synthetically display the images obtained by saidplurality of observation units of said observation section; and aspecifying section for specifying the images to be displayed accordingto the position and orientation of said observation section.
 23. Asurgical operation image acquisition/display apparatus comprising: anobservation section having a plurality of observation units and adaptedto modify its position and orientation; an image display section adaptedto display each of the images obtained by said plurality of observationunit of said observation section in a size either enlarged or reducedwith an independently selected magnitude; and a specifying section forspecifying the images to be synthetically combined according to theposition and orientation of said observation section.
 24. A surgicaloperation image acquisition/display apparatus comprising: an observationsection having a plurality of observation units and adapted to modifyits position and orientation; an image display section adapted todisplay an selected from the images obtained by said plurality ofobservation section of said observation section; and a specifyingsection for specifying the image to be selected according to theposition and orientation of said observation section.
 25. A surgicaloperation image acquisition/display apparatus comprising: an observationsection having a plurality of observation units and adapted to modifyits position and orientation relative to an area of surgical operation;an image display section adapted to synthetically display the imagesobtained by said plurality of observation units of said observationsection in a mode selected out of a plurality of different modes; and aspecifying section for specifying a mode of image synthesis to saidimage display section according to the position and orientation of saidobservation section.
 26. A surgical operation image acquisition/displayapparatus comprising: an observation section having a surgicalmicroscope adapted to modify its position and orientation and anendoscope also adapted to modify its position and orientation; adetection section for detecting the position and orientation of saidobservation section; an image display section adapted to syntheticallycombine the image observed by said surgical microscope and the imageobserved by said endoscope and display the images in a mode selected outof a plurality of different modes; and a specifying section forspecifying the mode of image synthesis to said image display sectionaccording to the outcome of the detection of said detection section. 27.A surgical operation image acquisition/display apparatus comprising: anobservation section having a surgical microscope adapted to modify itsposition and orientation and an endoscope also adapted to modify itsposition and orientation; a detection section for detecting the positionand orientation of said observation section; an image display sectionadapted to alternatively display the image observed by said surgicalmicroscope or the image observed by said endoscope; and a specifyingsection for specifying the image to be displayed according to theoutcome of the detection of said detection section.
 28. A surgicaloperation image acquisition/display apparatus according to claim 22 ,wherein said specifying section specifies in accordance with thedistance between the area of surgical operation and the observationsection.
 29. A surgical operation image acquisition/display apparatuscomprising: a first observation section for observing an area ofsurgical operation; a second observation section arranged apart fromsaid first observation section and adapted to observe at least said areaof surgical operation or the vicinity thereof; an observed imagesynthesizing section for synthetically combining the image observed bysaid first observation section and the image observed by said secondobservation section; a display section for display the image synthesizedby said observed image synthesizing section; a position and orientationdetection section for detecting at least either the position andorientation of said first observation section of that of said secondobservation section; and a state of synthesis modification specifyingsection for specifying the modification made to the state of synthesisof the observed images to said observed image synthesizing sectionaccording to the position and orientation information from said positionand orientation detection section.
 30. A surgical operation imageacquisition/display apparatus according to claim 29 , wherein said stateof synthesis modification specifying section specifies for use eitherthe image observed by said first observation section or the imageobserved by said second observation section according to the positionand orientation information of either said first observation section ofsaid second observation section fed from said position and orientationdetection section and either the relative distance between said area ofsurgical operation and said first observation section or the directionof said area of surgical operation as viewed from said secondobservation section.
 31. A surgical operation image acquisition/displayapparatus according to claim 29 , wherein said state of synthesismodification specifying section specifies reduction of either the imageobserved by said first observation section or the image observed by saidsecond observation section according to the position and orientationinformation of either said first observation section of said secondobservation section fed from said position and orientation detectionsection and either the relative distance between said area of surgicaloperation and said first observation section or the direction of saidarea of surgical operation as viewed from said second observationsection, and synthesis the reduced image into the other image.
 32. Anavigation apparatus comprising: navigation-related informationgenerating means for generating navigation-related information bymeasuring the relative position and orientation of an object and atarget in a three-dimensional space in order to navigate said object tosaid target; and display means for displaying said navigation-relatedinformation generated by said navigation-related information generatingmeans in different modes according to the relative position andorientation of said object and said target.
 33. A navigation apparatuscomprising: navigation-related information generating means forgenerating navigation-related information by measuring the relativeposition and orientation of an object and a target in athree-dimensional space in order to navigate said object to said target;and display means for displaying at least a model image of the object orthe target, information on the direction of navigation or information onthe distance between the object of navigation and the target when saidnavigation-related information generating means can measure saidposition and orientation in a three-dimensional space but displayinginformation indicating a measurement incapable situation when saidnavigation-related information generating means cannot measure saidposition and orientation.
 34. A navigation apparatus adapted to generateinformation for navigating an object to a target comprising:three-dimensional position and orientation measuring means for measuringthe three-dimensional position and orientation of said object and thatof said target; information generating means for generating informationnecessary for the navigation on the basis of the obtained result of saidthree-dimensional position and orientation measuring means; and displaymeans for displaying the information generated by said informationgenerating means; wherein said display means displays navigation-relatedinformation according to at least information on the distance betweensaid object and said target, information on the direction toward saidtarget as viewed from said object and information if either said objector said target is located within the effective area of measurement ofsaid three-dimensional position and orientation measuring means in adisplay mode selected out of a plurality of display modes.
 35. Anavigation apparatus comprising: a target of navigation; an object to benavigated to said target; three-dimensional position and orientationmeasuring means for measuring the three-dimensional position andorientation of at least said target or said object of navigation;computational information determining means for generatingnavigation-related information on the basis of the information obtainedby the measurement of said three-dimensional position and orientationmeasuring means and controlling the generated information; andinformation display means for display the navigation-related informationgenerated and controlled by said computational information determiningmeans; the relative distance between the target and the object ofnavigation and their orientations in a three-dimensional space aredetermined by the three-dimensional position and orientation measuringmeans; wherein said computational information determining means modifieseither the attribute or the type of said navigation-related informationas a function of at least the relative distance between said target andsaid object of navigation or the direction of said target as viewed fromsaid object measured by said three-dimensional position and orientationmeasuring means or the immeasurability thereof so as to make it visiblyreflect the outcome of measurement.
 36. A surgical operation imageacquisition/display apparatus comprising: observation means having aplurality of observation sections and adapted to modify its position andorientation; image display means adapted to alternatively orsynthetically display the images obtained by said plurality ofobservation sections of said observation means; and specifying means forspecifying the images to be displayed according to the position andorientation of said observation section.
 37. A surgical operation imageacquisition/display apparatus comprising: observation means having aplurality of observation sections and adapted to modify its position andorientation; image display means adapted to display each of the imagesobtained by said plurality of observation sections of said observationsection in a size either enlarged or reduced with an independentlyselected magnitude; and specifying means for specifying the images to besynthetically combined according to the position and orientation of saidobservation means.
 38. A surgical operation image acquisition/displayapparatus comprising: observation means having a plurality ofobservation sections and adapted to modify its position and orientation;image display means adapted to display an selected from the imagesobtained by said plurality of observation sections of said observationmeans; and specifying means for specifying the image to be selectedaccording to the position and orientation of said observation means. 39.A surgical operation image acquisition/display apparatus comprising:observation means having a plurality of observation means and adapted tomodify its position and orientation relative to an area of surgicaloperation; image display means adapted to synthetically display theimages obtained by said plurality of observation means of saidobservation section in a mode selected out of a plurality of differentmodes; and specifying means for specifying a mode of image synthesis tosaid image display means according to the position and orientation ofsaid observation section.
 40. A surgical operation imageacquisition/display apparatus comprising: observation means having asurgical microscope adapted to modify its position and orientation andan endoscope also adapted to modify its position and orientation; adetection means for detecting the position and orientation of saidobservation means; image display means adapted to synthetically combinethe image observed by said surgical microscope and the image observed bysaid endoscope and display the images in a mode selected out of aplurality of different modes; and specifying means for specifying themode of image synthesis to said image display means according to theoutcome of the detection of said detection means.
 41. A surgicaloperation image acquisition/display apparatus comprising: observationmeans having a surgical microscope adapted to modify its position andorientation, and an endoscope also adapted to modify its position andorientation; detection means for detecting the position and orientationof said observation means; image display means adapted to alternativelydisplay the image observed by said surgical microscope or the imageobserved by said endoscope; and specifying means for specifying theimage to be displayed according to the outcome of the detection of saiddetection means.
 42. A surgical operation image acquisition/displayapparatus comprising: first observation means for observing an area ofsurgical operation; second observation means arranged apart from saidfirst observation means and adapted to observe at least said area ofsurgical operation or the vicinity thereof; observed image synthesizingmeans for synthetically combining the image observed by said firstobservation means and the image observed by said second observationmeans; display means for display the image synthesized by said observedimage synthesizing means; position and orientation detection means fordetecting at least either the position and orientation of said firstobservation means of that of said second observation means; and state ofsynthesis modification specifying means for specifying the modificationmade to the state of synthesis of the observed images to said observedimage synthesizing means according to the position and orientationinformation from said position and orientation detection means.